feat(logger): new stack_trace field with rich exception details #3147

stevrobu · 2023-09-28T20:18:12Z

Issue number: #1268

Summary

Changes

Added environment variable and formatter constructor parameter to allow users to include additional JSON formatted full stack trace to their logger.exception output.

To test the change, the following Lambda code will enable the enhanced stacktrace and log the exception along with the stacktrace:

import requests

from aws_lambda_powertools import Logger
from aws_lambda_powertools.utilities.typing import LambdaContext

ENDPOINT = "http://httpbin.org/status/500"
logger = Logger(serialize_stacktrace=True)


def lambda_handler(event: dict, context: LambdaContext) -> str:
    try:
        ret = requests.get(ENDPOINT)
        ret.raise_for_status()
    except requests.HTTPError as e:
        logger.exception(e)
        raise RuntimeError("Unable to fullfil request") from e

    return "hello world"

Response in CloudWatch logs should be:

{
   "level":"ERROR",
   "location":"lambda_handler:16",
   "message":"500 Server Error: INTERNAL SERVER ERROR for url: http://httpbin.org/status/500",
   "timestamp":"2023-10-09 17:47:50,191+0000",
   "service":"service_undefined",
   "exception":"Traceback (most recent call last):\n  File \"/var/task/app.py\", line 14, in lambda_handler\n    ret.raise_for_status()\n  File \"/var/task/requests/models.py\", line 1021, in raise_for_status\n    raise HTTPError(http_error_msg, response=self)\nrequests.exceptions.HTTPError: 500 Server Error: INTERNAL SERVER ERROR for url: http://httpbin.org/status/500",
   "exception_name":"HTTPError",
   "stack_trace":{
      "type":"HTTPError",
      "value":"500 Server Error: INTERNAL SERVER ERROR for url: http://httpbin.org/status/500",
      "module":"requests.exceptions",
      "frames":[
         {
            "file":"/var/task/app.py",
            "line":14,
            "function":"lambda_handler",
            "statement":"ret.raise_for_status()"
         },
         {
            "file":"/var/task/requests/models.py",
            "line":1021,
            "function":"raise_for_status",
            "statement":"raise HTTPError(http_error_msg, response=self)"
         }
      ]
   }
}

User experience

User can now parse JSON stacktraces instead of string stacktraces.

Checklist

If your change doesn't seem to apply, please leave them unchecked.

Meet tenets criteria
I have performed a self-review of this change
Changes have been tested
Changes are documented
PR title follows conventional commit semantics

Is this a breaking change?

Not a breaking change.

RFC issue number:

Checklist:

Migration process documented
Implement warnings (if it can live side by side)

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

boring-cyborg · 2023-09-28T20:18:16Z

Thanks a lot for your first contribution! Please check out our contributing guidelines and don't hesitate to ask whatever you need.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

leandrodamascena

Hi @stevrobu! Thanks for submitting this PR, this is an amazing first contribution! We need to change a few things before merging this.

One thing I'm missing is the include_stacktrace parameter in the Logger instance. I see you added it in PowertoolsFormatter, but I think we can improve the user experience by adding the Logger class in the same way. You can keep this parameter in PowertoolsFormatter because someone can use the Formatter class in some special cases, but adding this in the Logger class improves the experience.

from aws_lambda_powertools import Logger

logger = Logger(include_stacktrace=True)

aws_lambda_powertools/logging/formatter.py

tests/functional/test_logger.py

aws_lambda_powertools/logging/formatter.py

stevrobu · 2023-10-03T16:03:39Z

@leandrodamascena : I will add the parameter to the logger as you suggested. It actually already works that way because any of the kwargs from the Logger class initialization are passed to the formatter in the structure_logs method. I actually didn't realize this until I wrote a test prior to making the update and the test passed.

I've updated the tests so that the linter does not complain and I'm checking out the other suggestions.

leandrodamascena · 2023-10-03T21:58:19Z

@leandrodamascena : I will add the parameter to the logger as you suggested. It actually already works that way because any of the kwargs from the Logger class initialization are passed to the formatter in the structure_logs method. I actually didn't realize this until I wrote a test prior to making the update and the test passed.

I've updated the tests so that the linter does not complain and I'm checking out the other suggestions.

Hi @stevrobu! I think you need to push your local commits, right? I can't see the new code.

stevrobu · 2023-10-03T22:02:26Z

I have not finished yet. I should be able to push tomorrow. On Oct 3, 2023 5:58 PM, Leandro Damascena ***@***.***> wrote: @leandrodamascena<https://github.com/leandrodamascena> : I will add the parameter to the logger as you suggested. It actually already works that way because any of the kwargs from the Logger class initialization are passed to the formatter in the structure_logs method. I actually didn't realize this until I wrote a test prior to making the update and the test passed. I've updated the tests so that the linter does not complain and I'm checking out the other suggestions. Hi @stevrobu<https://github.com/stevrobu>! I think you need to push your local commits, right? I can't see the new code. — Reply to this email directly, view it on GitHub<#3147 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A35SEV6LDS47Z52LSODPJRLX5SDAPAVCNFSM6AAAAAA5LQEGF6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONBVG44TCMJRHE>. You are receiving this because you were mentioned.Message ID: ***@***.***>

leandrodamascena · 2023-10-03T22:05:23Z

I have not finished yet. I should be able to push tomorrow.

Aaa, sorry for my mistake. I saw you resolve conversations and assumed you made the commits. No rush, take your time.

stevrobu · 2023-10-06T14:57:15Z

@leandrodamascena - I have checked in the latest updates. Thanks for the feedback!

…da-python into develop

codecov-commenter · 2023-10-09T17:27:46Z

Codecov Report

All modified lines are covered by tests ✅

Comparison is base (e72fbbe) 95.95% compared to head (8c1831e) 95.96%.
Report is 6 commits behind head on develop.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #3147   +/-   ##
========================================
  Coverage    95.95%   95.96%           
========================================
  Files          195      195           
  Lines         8364     8381   +17     
  Branches      1559     1562    +3     
========================================
+ Hits          8026     8043   +17     
  Misses         276      276           
  Partials        62       62

Files	Coverage Δ
aws_lambda_powertools/logging/formatter.py	`100.00% <100.00%> (ø)`
aws_lambda_powertools/logging/logger.py	`99.45% <ø> (ø)`
aws_lambda_powertools/logging/types.py	`100.00% <100.00%> (ø)`

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

leandrodamascena

Hey @heitorlessa can you pls review this PR?

For me, it is approved! Amazing work @stevrobu! 🚀

rubenfonseca · 2023-10-10T09:29:26Z

Looking at this now

rubenfonseca

Left some comments and questions, but overall looks good!

docs/core/logger.md

docs/contributing/documentation/rfcs.md

sonarqubecloud · 2023-10-10T11:38:25Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information

No need

boring-cyborg · 2023-10-10T11:48:11Z

Awesome work, congrats on your first merged pull request and thank you for helping improve everyone's experience!

stevrobu added 2 commits September 26, 2023 16:32

Added optional stack trace to exception logging. Issue 1268

163b908

Added tests and documentation

3ad3c76

stevrobu requested a review from a team September 28, 2023 20:18

boring-cyborg bot added commons logger labels Sep 28, 2023

pull-request-size bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Sep 28, 2023

Added tests and documentation.

5436377

boring-cyborg bot added documentation Improvements or additions to documentation tests labels Sep 29, 2023

pull-request-size bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Sep 29, 2023

leandrodamascena linked an issue Oct 2, 2023 that may be closed by this pull request

Feature request: Parse exceptions into structured logs #1268

Closed

2 tasks

Merge branch 'develop' into develop

dda7593

leandrodamascena mentioned this pull request Oct 2, 2023

Feature request: Parse exceptions into structured logs #1268

Closed

2 tasks

leandrodamascena changed the title ~~feat(logger): Parse exceptions into structured logs~~ feat(logger): add structured exception logging in JSON format Oct 2, 2023

Merge branch 'develop' into develop

c57a77b

leandrodamascena requested changes Oct 2, 2023

View reviewed changes

dreamorosi reviewed Oct 3, 2023

View reviewed changes

aws_lambda_powertools/logging/formatter.py Outdated Show resolved Hide resolved

Updated the tests, serialization method and renamed the parameter

faf838f

boring-cyborg bot added the internal Maintenance changes label Oct 6, 2023

stevrobu and others added 4 commits October 6, 2023 13:34

Removed dead code

fee5479

Merge branch 'develop' into develop

2a47946

Removed env variable and default stack trace logging to True

6d5f148

Merge branch 'develop' of https://github.com/stevrobu/powertools-lamb…

0b79f69

…da-python into develop

leandrodamascena added 3 commits October 9, 2023 17:32

Small changes

2ddfde9

Removing wrong field

9307e9d

Adjusting tests

8eba534

leandrodamascena added 2 commits October 9, 2023 19:42

Merge branch 'develop' into develop

b7294cb

Adjusting documentation

ed2a3c5

leandrodamascena requested review from heitorlessa and leandrodamascena October 9, 2023 17:59

leandrodamascena previously approved these changes Oct 9, 2023

View reviewed changes

rubenfonseca self-requested a review October 10, 2023 09:29

rubenfonseca suggested changes Oct 10, 2023

View reviewed changes

docs/core/logger.md Outdated Show resolved Hide resolved

docs/contributing/documentation/rfcs.md Show resolved Hide resolved

leandrodamascena added 2 commits October 10, 2023 11:28

Merge branch 'develop' into develop

e2bc82e

Addressing Ruben's feedback

d24a566

leandrodamascena dismissed their stale review via d24a566 October 10, 2023 10:32

leandrodamascena requested a review from rubenfonseca October 10, 2023 10:36

rubenfonseca previously approved these changes Oct 10, 2023

View reviewed changes

Addressing Ruben's feedback

8c1831e

leandrodamascena dismissed rubenfonseca’s stale review via 8c1831e October 10, 2023 11:37

heitorlessa changed the title ~~feat(logger): add structured exception logging in JSON format~~ feat(logger): new stack_trace field with rich exception details Oct 10, 2023

heitorlessa approved these changes Oct 10, 2023

View reviewed changes

leandrodamascena self-requested a review October 10, 2023 11:47

leandrodamascena previously approved these changes Oct 10, 2023

View reviewed changes

leandrodamascena merged commit e555df1 into aws-powertools:develop Oct 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(logger): new stack_trace field with rich exception details #3147

feat(logger): new stack_trace field with rich exception details #3147

stevrobu commented Sep 28, 2023 •

edited by leandrodamascena

Loading

boring-cyborg bot commented Sep 28, 2023

leandrodamascena left a comment

stevrobu commented Oct 3, 2023

leandrodamascena commented Oct 3, 2023

stevrobu commented Oct 3, 2023 via email

leandrodamascena commented Oct 3, 2023

stevrobu commented Oct 6, 2023

codecov-commenter commented Oct 9, 2023 •

edited

Loading

leandrodamascena left a comment

rubenfonseca commented Oct 10, 2023

rubenfonseca left a comment

sonarqubecloud bot commented Oct 10, 2023

boring-cyborg bot commented Oct 10, 2023

feat(logger): new stack_trace field with rich exception details #3147

feat(logger): new stack_trace field with rich exception details #3147

Conversation

stevrobu commented Sep 28, 2023 • edited by leandrodamascena Loading

Summary

Changes

User experience

Checklist

Acknowledgment

boring-cyborg bot commented Sep 28, 2023

leandrodamascena left a comment

Choose a reason for hiding this comment

stevrobu commented Oct 3, 2023

leandrodamascena commented Oct 3, 2023

stevrobu commented Oct 3, 2023 via email

leandrodamascena commented Oct 3, 2023

stevrobu commented Oct 6, 2023

codecov-commenter commented Oct 9, 2023 • edited Loading

Codecov Report

leandrodamascena left a comment

Choose a reason for hiding this comment

rubenfonseca commented Oct 10, 2023

rubenfonseca left a comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Oct 10, 2023

boring-cyborg bot commented Oct 10, 2023

stevrobu commented Sep 28, 2023 •

edited by leandrodamascena

Loading

codecov-commenter commented Oct 9, 2023 •

edited

Loading